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METHOD OF ORGANIZING DIGITAL IMAGES 



ON A PAGE 




FIELD OF THE INVENTION 



Qy^\ The present invention relates to a method, system and computer 
5 software program for automatically organizing digital on a page that is especially 
useful when images of various sizes are presented for placement on the page. 



pictorial information. Typically this pictorial information corresponds to 

10 important life memories of the individual who created the photographic album. 
These memories can correspond to photographs, and keepsakes and mementos 
such as, greeting cards, invitations, hand drawn images, etc. Each year an 
extremely large number of photographs are developed, looked at and stuffed in a 
drawer or shoebox. Digital images suffer the same fate as their hardcopy cousins 

15 except in this case they are "stuffed onto" digital storage devices such as hard 
drives and CD-ROMS. The individual usually has good intentions of creating a 
photo album, but never quite gets aroimd to it. This is because the process of 
creating a photo album is a time consuming, difficult deed that usually provides 
less than satisfactory results. 

20 Typically the process of creating a photo album is done by either 

cutting and attaching hardcopy images onto an albimi page or by scaling, 
cropping, and inserting digital images into album pages via commercially 
available imaging software such as Microsoft Picturelt™. One important step in 
creating an album page is the ability to easily arrange and fit the desired nimiber 

25 of digital images on the album page. One way of simplifying the process of 

arranging digital images is to use templates that have predefined locations for the 
images. Templates provide an easy way to album images but they are not always 
efficient in their use of space. With templates the selected images are either 
proportionately sized or cropped and sized to fit in the selected location in the 

30 template. Proportionately sizing the image to fit a template can lead to 

considerable space being wasted on the albimi page. The process of cropping an 
image to fit a template can lead to important pictorial information being cut out of 




BACKGROUND OF THE INVENTION 



Photographic albums provide a way to store, organize, and display 




-2- 



the image. A further disadvantage to templates is the user can only choose from 
the available templates. If there are no acceptable templates available, the user is 
forced to resort to the tedious process of individually scaling, cropping, and 
placing the images on the page until an acceptable arrangement is foimd. With the 
5 advent of Advanced Photo System, digital scanners, digital cameras, and digital 
image editing software, digital images can have a multitude of various aspect 
ratios. This can make the process of arranging images on a page even more 
challenging. In addition, it is impossible to have templates that can facilitate all 
the possible combination of images. 

10 The present invention solves many of the problems of the prior art. 

It provides an easy and efficient way of arranging digital images of various size 
aspect ratios on an album page. In addition, the current invention provides the 
user with an easy way to generate a large number of different arrangements of the 
same images and thus allowing the user to choose a preferred arrangement. 

15 SUMMARY OF THE INVENTION 

The above, and other objects, advantages and novel features of the 
present invention will become more apparent from the accompanying detailed 
description thereof when considered in conjunction with the following drawings. 

In accordance with one aspect of the present invention there is 

20 provided a method of organizing a plurality of images in a predetermined page 
format, comprising the steps of: 

grouping the plurality of images into a plurality of different page 

layouts; 

analyzing each of the different page layouts in accordance with a 
25 predetermined criteria; and 

selecting the page layout based on the predetermined criteria. 
In accordance with another aspect of the present invention there is 
provided a system for organizing a plurality of images in a predetermined format, 
comprising: 

30 a first computer for composing a plurality of digital images on a 

page; 
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a software program such that when loaded on the computer will 
cause the computer to group the plurality of images into a plurality of different 
page layouts; 

analyzing each of the different page layouts in accordance with a 
5 predetermined criteria; and 

selecting the page layout based on the predetermined criteria. 
In accordance with another aspect of the present invention there is 
provided a computer software product for organizing a plurality of images in a 
predetermined format comprising a computer readable storage medium having a 
10 computer program which when loaded into a computer causes the computer to 
perform the following steps: 

grouping the plurality of images into a plurality of different page 

layouts; 

analyzing each of the different page layouts in accordance with a 
15 predetermined criteria; and 

selecting the page layout based on the predetermined criteria. 
In accordance with yet another aspect of the present invention there 
is provided a method of organizing a plurality of images in a predetermined page 
format, comprising the steps of: 
20 providing a plurality of digital images; 

selecting a number of the images for placement on the 
predetermined format; 

grouping the plurality of images into a plurality of different page 

layouts; 

25 analyzing each of the different page layouts in accordance with a 

predetermined criteria; and 

selecting the page layout based on the predetermined criteria. 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the detailed description of the preferred embodiments of the 
30 invention presented below, reference is made to the accompanying drawings in 
which: 
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Fig.l is a schematic diagram of a system for practicing the present 

invention; 

Fig. 2 is a page having a template layout in accordance with the 

prior art; 

5 Fig. 3 is plan view of a plurality of images for placement on an 

page; 

Fig. 4 is a plan view of the page of Fig.2 having the images of Fig. 
3 placed thereon; 

Fig. 5 flow chart of a method for determining a page layout for a 
10 plurality of images in accordance with the present invention; 

Fig. 6 is plan view of the format of a page on which the images of 
Fig. 3 are to be placed in accordance with the present invention; 

Fig. 7 is a flow chart illustrating one of the steps in calculating the 
page layout method of Fig. 5; 
15 Fig. 8 is an initial image page layout of the images to be placed on 

the page; 

Fig. 9 is a modified image page layout of Fig. 8 after fiirther 

manipulation; 

Fig. 10 is a modified image page layout of Fig. 9 after yet still 
20 fiirther manipulation; 

Figs. 11-15 illustrate yet fiirther manipulation that may be made to 
the image page layout; and 

Fig. 16 is flow chart of a method of providing fiirther image page 

manipulation. 

25 DETAILED DESCRIPTION OF THE INVENTION 

In the present invention, the term page as used herein is meant to 
include an album or scrapbook page, poster, soft copy display, or any other format 
where images are displayed. 

FIG. 1 illustrates a system 10 that can be usefiil in practicing the 
30 present invention. The system 10 includes a personal computer PC 12 containing 
a central processing unit (CPU) that can execute a set of predefined steps in 
carrying out the method of the present invention. A digital storage media 20 is 
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also provided in PC 12 for storing digital images. The digital storage media 20 

can include different types of devices, such as RAM, ROM, hard and floppy 

drives, etc. The digital storage media 20 can also be used to store the generated 

image page. In addition, digital image capture devices such as scanner 28 and 

5 digital cameras 30, which are additional sources of digital images, can also be 

provided to the computer 12. It is to be understood that the digital images may be 

obtained from any source. A user interacts with the computer 12 via input devices 

40, such as a mouse and/or keyboard, and a display monitor 50 that is connected 

to the computer 12. The system 10 may also contain a device such as a printer 61 

10 for outputting the image page. Altematively, the above components do not have 

lii to all reside on the host computer 12 but can reside on a server 63 located at a 

\% network service provider 64 which can be connected via a communication 

network 70. The commimication network 70 may comprise the Internet 74 which 

1= can be accessed by an individual using an Intemet Service Provider (ISP) 76. The 

■3. 1 ^ 

15 network service provider 64 may also include a customer database 66 for storing 

;L,^ information regarding customers and a image storage data base for storing of 

.41 digital images provided by the customer either by the intemet 74 or by any other 

•Ji means. The service provider 64 will be equipped to provide goods and/or services 

as described herein or any other goods and/or services desired. The remote 

20 network service provider may also be accessed by a customer using a retail kiosk 

or any other appropriate commimication device. 

Referring to Fig. 2 there is illustrated a prior art, a page 41 that 

incorporates a template 42 made in accordance with the prior art. The template 42 

comprises a plurality of predefined locations 43 - 47 for the placement of images 

25 that are used to simplify the process of arranging images on a page. A weakness 

of using pages with a predefined template becomes very apparent when the 

images selected to be arranged on the page have very diverse aspect ratios. FIG. 3 

shows five images 54, 56, 58, 60, and 62 that are to be arranged on a page and 

FIG. 3 shows a page that has predefined locations for the five images. FIG. 4 

30 shows the results of proportionately sizing the images to fit in template 42. As 

can be seen, this leads to a considerable amoimt of space on the page being 
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squandered. That is, large areas of the page are void of images. In addition the 
size of the images are substantially reduced when they are sized to fit the location. 

FIG. 5 is a schematic flow chart illustrating the method of 
constructing an image page layout according to the present invention using 
5 computer software program made in accordance with the present invention. A 
pliu:*ality of digital images (such as shown by FIG. 3) that can be placed on an 
image page are stored in a database at step 100. In the present invention, digital 
images refers not only to images obtained fi-om photographs, but to digital images 
obtained fi-om any source, for example but not by way of limitation, digital 

10 camera, scanning of a hard copy document, or electronically from another source. 
The format of a page 80 (see FIG. 6) on which the images 54, 56, 58, 60, and 62 
are to be arranged is selected at step 110 by either specifying the height and width 
of the page oi- alternatively the eispect ratio (i.e. width/ height or height/width) of 
the page can be specified. The digital images to be arranged on the image page 

15 are then selected at step 120 either manually, semi-automatically, or automatically 
from the database where the images are stored. The semi-automatic and 
automatic selection process, can make use of data, such as time and date, that is 
recorded along with images and digital image processing techniques, such as 
image content analysis, to help select relevant images that should appear on a 

20 given image page. Optionally, the images 54, 56, 58, 60, and 62 to be arranged on 
the page can be normalized at step 130. Normalizing the images prevents one 
image from spatially dominating the page layout. This is especially true when one 
image is much larger or smaller than the rest of the images. A preferred 
normalization is one in which the images are isotropically scaled so that their 

25 shortest dimension (height or width) are all equivalent. The normalized height 
and width of the images to be arranged on the image page and the format of the 
image page are passed to the page layout subroutine 140. The layout subroutine 
140 calculates a page layout of the images on the image page 80 and displays 150 
the results on display monitor 50. At this point the user can either accept 160 the 

30 image page layout or iterate through the page layout subroutine 140 until an 
acceptable image page is obtained. When an acceptable image page layout is 
obtained at step 170, the image page layout and images may be stored on the host 
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computer 12, printed by printer 61, or transmitted to another site via the network 
70 where it can be stored, printed or view by another person. 

The job of the page layout subroutine 140 is to fit a given nximber n 
of images on a given image page to obtain a suitable page layout using a 
5 predetermined criteria. In the embodiment illustrated the predetemiined criteria 
comprised fitting the n images onto the image page in such a way that minimizes 
the white space. The white space, which is the area of the image page not covered 
by an image, is defined by the following relationship: 

EQ. #1 White Space = 1.0- (total image area)/(page area) 
10 Wherein: the total image area is the sirni of the area of the 

individual images to be placed on the page; and 

the page area is the area as defined by the height H and Width W 
allowed for placement of the images. Generally, the page area would be the entire 
page. 

15 The page layout subroutine 140 may take into account the aesthetic 

considerations of the image page layout. One important aesthetic consideration is 
the spatial balance of the images on the page. Spatial balance is a measure of how 
equally distributed the images are on the image page. 

The problem of generating an acceptable image page layout that 

20 contains n images can be formulated as a combinatorial optimization problem. 
The most practical way of solving combinatorial optimization problems is to use 
stochastic algorithms, such as simulated annealing or genetic algorithms. These 
combinatorial optimization algorithms along with other optimization algorithms 
are described in Iterative Computer Algorithms with Applications in Engineering: 

25 Solving Combinatorial Optimization Problems by Sait, S. M. and Youssef, H., 
IEEE Computer Society, Los Alamitos, Calif , 1999. 

Referring to FIG. 7, there is shown a preferred embodiment of a 
flow chart for a page layout subroutine 140 for calculating the layout of the 
images on a page according to the present invention. The subroutine 140 in the 

30 embodiment illustrated is automatically executed by a computer software program 
that is separate or part of a larger software program for executing the method of 
Fig. 5. It is to be imderstood that the page layout subroutine 140 may be executed 
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by a variety of alternative methods and optimization techniques. The height and 
width of the n images to be arranged on the page and the format of the page are 
passed to the page layout subroutine 140. A trail page layout 212 is generated that 
contains all n images such that no two images overlap. 
5 In the embodiment illustrated, two useful methods that can each be 

used for generating non-overlapping placement of the images on a page, are the 
sequence pair and boimded sliceline grid BSG structures. The sequence pair 
structure is described in H. Murata, K. Fujiyoshi, S. Nakatake, and Y. Kajitani, 
"VLSI subroutine placement based on rectangle-packing by the sequence pair," 

10 IEEE Trans, Computer-Aided Design, vol. 15, pp. 1518-1524, Dec. 1996 and the 
BSG structure is described in S. Nakatake, K. Fujiyoshi, H. Murata, and Y. 
Kajitani, "Subroutine placement on BSG-structure and IC layout applications," 
Proc. lEEEInL Conf. Computer-Aided Design, pp. 484-491, 1996. 

At step 200, the trail page layout 212 is generated by using one of 

15 the above methods to produce a non-overlapping placement of the n images on a 
plane and then isotropically scaling (i.e. the width and height are scaled by the 
same proportion) each image by the same proportion so as to fit onto the chosen 
page format. FIG. 8 illustrates a trail page layout 212 that contains the five 
images shown in FIG. 3. The trail page layout 212 is then scored by at step 210 by 

20 a cost (or objective) function. In the preferred embodiment the cost fimction is 
equal to the white space as defined by EQ. #1 . 

The goal of the optimization is to find a page layout that minimizes 
the cost fimction. In the preferred embodiment a simulated annealing approach is 
used to find an optimal page layout. For simulated annealing, a modification or 

25. change is made to the current trial page layout 212 to generate a new trial page 
layout 222 (see Fig. 9) at step 220. New trail page layout 222 is generated by 
randomly changing the relative positions of the images in the current trail page 
layout 212. The new trail page layout 222 is then scored at step 230 by the cost 
fimction. FIG. 9 illustrates the new trial page layout 222 that was generated by 

30 randomly perturbing the page layout shown in FIG. 8. From herein the prior trail 
page layout refers to the trail page layout that was modified to generate the next 
new trail page layout. 
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A determination is made as whether to keep the new trail page 



layout 222 or the prior trail page layout 212 at step 240. The new trial page layout 
222 is accepted if its score is lower than the score of the prior trial page layout 
212, Also, if the score for the new page layout 222 is greater than the score the 
5 prior page layout 212, the new page layout 222 is kept with a defined probability 
of exp[-(Ascore)/T] where Ascore = (score of new page layout — source of prior 
page layout). The parameter T is used to adjust the probability of keeping a new 
page layout. The optimization process is iterated at steps 220, 230, 240, and 250 
until the last iteration is obtained. The last iteration can be defined by the total 
10 number of iterations or by a number of iterations without an improvement. It is to 
be understood that any desired criteria may be used for determining when no 
further iteration is needed or desired. 



be made by fiirther scaling the images by different amoxmts so to fiirther minimize 

15 the white space. Once again a simulated annealing approach may be used. A new 
trail page layout is generated by randomly selecting an image and scaling it by a 
random factor between a minimum factor scale 1 and a maximum factor scale2 
while respecting the relative positions of the images imposed by the page layout 
calculated by iterating through steps 220, 230, 240 and 250. Scale J represents the 

20 lower scaling limit and scale2 represents the upper scaling limit. By varying the 
scale limits scalel and scalel^ page layouts can be generated that possess very 
different artistic look and feel. A small magnitude difference between the scaling 
factors scalel and scalel (e.g. scalel = 0.9 and scale2 =1.1) ensures that no one 
image will overwhelm another image on the page. A large magnitude difference 

25 between scalel and scale2 can lead to optimized page layouts with very little 

white space where the images have very diverse sizes. A new page layout 272 is 
scored and compared to the prior page layout to determine which page layout to 
keep. Through an iterative process of modifying, scoring, and comparing a new 
page layout 272 such as shown in FIG. 10 is determined at step 260. In the new 

30 optimum page layout of Fig. 10 was generated by scaling each image in the page 
layout shown in FIG. 9 by random factors between 0.9 - 1.1. 



Optionally, further improvements in the calculated page layout can 
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in a further optionally step 270, the images in the page layout can 
be aesthetically balanced. Aesthetic considerations play a role determining what 
makes a visually appealing page layout to an individual. There are many different 
artistic looks and feels that a page layout can possess and is very personal. In a 
5 preferred embodiment, the images are aesthetically balanced by positioning them 
on the page so that they are equally distributed on the page with visually appealing 
borders between each image and the border of the page while still preserving the 
amount of white space. For example, but not by way of limitation, the images are 
positioned on the page such that the left and right borders (space) 273, 274 

10 between an image and the next closest image or boundary (peripheral edge) of the 
page are made equal as shown in FIG. 12. The same is done for the top and 
bottom borders 275, 277 of the images as shown in FIG. 12. Note that the 
left/right 273, 274 and bottom/top 275, 277 borders of the images do not 
necessarily have to be equal. In addition, images that are close to the boundary of 

15 the page can be made left and/or right and/or top and/or bottom justified. 

Referring to FIG. 16, there is shown a flow chart of an embodiment 
for spatially balancing the images and equalizing the top/bottom and right/left 
borders of the images on a page according to the present invention. It is of course 
imderstood that the process of equalizing the borders/spaces around an image may 

20 be executed by a variety of altemative methods and optimization techniques. The 
process of generating equal top/bottom and equal left/right borders around each 
image in the page layout is an iterative process where the vertical and horizontal 
spacing between the image are determined independently. At step 300, the n 
images are sorted according to their vertical (y) axis 302 (see Fig. 11). Next at 

25 step 310 the bottommost image is set to the current image. The spaces between 
the closest image or page boundary above and below the current image are 
determined at step 320. At step 330, the current image is moved vertically so that 
the spaces (borders) between itself and the image or page boimdary above and 
below are equal. At step 340, the closest image above the current image is set as 

30 the new current image. Steps 320, 330, 340 and 350 are repeated imtil the topmost 
image has been repositioned. The analogous process is repeated at steps 360, 370, 
380, 390, 400 and 410 for the horizontal spacing, as shown by arrow 304 in Fig. 
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11, between the images. The steps 300, 310, 320, 330, 340, 350, 360, 370, 380, 
390, 400, 410 and 420 are then iterated until the images have reached their 
equilibrium position. That occurs when none of images are repositioned through a 
single iteration of steps 300, 310, 320, 330, 340, 350, 360, 370, 380, 390, 400 and 
5 410. At this point images are spatially balanced and the borders between the 
images have been equalized. 

FIG. 1 1 illustrates a page layout that was generated by equally 
distributing the images in the page layout 276 shown in FIG. 10. As shown in 
FIG. 1 1 some of the images are either touching each other and do not have white 
10 space between them or touching the boundary of the page. To ameliorate this 
iri look, the images can be scaled down and centered in their current locations. FIG. 

]% 12 illustrates a page layout 278 where the images in FIG. 1 1 have been scaled 

down by a factor of 0.95 and centered in their current locations. 
j= To generate a more whimsical or scrapbook look the images can be 

15 randomly rotated or rotated in a predetermined pattem. The rotation of the images 
should be constraint such that the rotated images do not overlap or have a 
maximum overlap so as not to obscure important detail in the images. For 
J;^ example, no overlapping image which is greater than 10% or that avoids covering 

i^j the central area of the image. FIG. 13 shows a page layout 280 where the images 

20 in page layout shown in FIG. 12 have been randomly rotated between -5 and +5. 

It is to be understood that various other modifications may be 
made. For example, but not by way of limitation, both the placement and scaling 
of the images can simultaneously be optimized in the calculation of the image 
page layout. The cost function can be generalized to contain many different goals 
25 in determining the page layout. Thus, in a further embodiment, the cost fimction 
can also take into account the aesthetic value in the calculation of the image page 
layout. In yet another embodiment of the invention, the optimization can 
minimize white space while simultaneously trying to avoid placing any images in 
a predefined location on the image page. FIG. 14 illustrates a page layout 282 
30 where the five images 54, 56, 58, 60, and 62 were placed so as not to be in the 
predefined location 283 represented by the cross hatched region. While in the 
embodiment illustrated by FIG. 14, the location 283 is in the lower right area. 
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However, location 283 may be anywhere on the page. For example, but not by 
way of limitation, location 283 may be in the center of the page with the images 
surroimding the location 283. This maybe especially useful when an image or 
artwork to be placed in the central location is related to the other images. In yet 
5 another embodiment, the location of a given image can be constrained to appear in 
a predefined location on the page layout. Li yet another embodiment, the scaling 
factors used to scale the images can be constrained in such a way as to emphasis a 
given image relative to the other images. FIG. 15 illustrates a page layout 284 
containing the five images 54, 56, 58, 60, and 62 were the scaling factors were set 

10 so as to emphasis image 58 over images 54, 56, 60, and 62. 

Optimization techniques try to find the global minimum, for 
example, the image layout that has the lowest possible amount of white space, but 
they are susceptible to being trapped in a local minimum. Since the process of 
optimizing the page layout is a random process, different optimal page layouts can 

15 be generated each time the same images and page format are run through the page 
layout subroutine 140. The system can automatically iterate through the page 
layout subroutine generating a predefined nimiber of image page layouts. The 
page layout that has the highest score is then chosen as the preferred page layout. 
Though it should be noted that the most aesthetically pleasing page layout might 

20 not correspond to the page layout that has the minimmn amount of white space. 

The invention has been described in detail with particular reference 
to certain preferred embodiments thereof, but it will be imderstood that variations 
and modifications can be effected within the spirit and scope of the invention. 
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